1. Field of the Invention
The present invention relates to an image processing apparatus and image processing method for converting color image data inputted from an image input apparatus to color space which is not dependent on the apparatus and/or lighting with a high precision.
2. Prior Art
In recent years, a scanner, digital camera, printer, display, and other various apparatuses have been utilized as an apparatus for processing a color image. As a technique for changing image data among these apparatuses, there is a technique comprising once converting color image data inputted from an input apparatus to an independent color space which is not dependent on the apparatus, and converting the space to the color image data to be outputted to an output apparatus. When conversion of a signal of the image input apparatus and the color space not dependent on the apparatus is established in this manner, the data can be transferred to any image output apparatus, and therefore it is unnecessary to set the same number of color converting processings as the number of combinations of the input apparatus and output apparatus. Moreover, when the color image data inputted from the image input apparatus is converted to the independent color space not dependent on not only the apparatus but also lighting, an image under lighting different from lighting of a time of image input can also be outputted via the output apparatus.
It is general to use XYZ three stimulus values defined by International Standardization Organization CIE, L*a*b* color specification system, L*u*v* color specification system, CAM 97s or another color appearance model as the independent color space which is not dependent on the apparatus. An attribute value of the color appearance model is calculated from the XYZ three stimulus values. Therefore, when the XYZ three stimulus values can be estimated from the signal of the image input/output apparatus, the color conversion is possible. Moreover, it is general to use spectral reflectance of a subject as the color space which is not dependent on the apparatus and lighting. When the spectral reflectance is integrated with desired lighting, the XYZ three stimulus values can be calculated. To estimate the XYZ three stimulus values or the spectral reflectance of the subject (object) from the color space of each image input/output apparatus is called characterization. The present invention relates to the characterization of the image input apparatuses such as a digital camera, multi-spectral camera, and scanner.
Examples of a conventional characterization technique of the image input apparatus include a method of measuring color of skin and method of estimating reflection spectrum described in Japanese Patent Application Laid-Open No. 174631/1995, a color reproduction apparatus described in Japanese Patent Application Laid-Open No. 85952/1999, and a color simulation apparatus described in Japanese Patent Application Laid-Open No. 233490/1997.
In the Japanese Patent Application Laid-Open No. 174631/1995, a method of estimating the reflection spectrum of the skin from the image inputted from the image input apparatus is disclosed. The procedure will be described with reference to FIG. 31. First, image data RGB inputted in procedure 3101 is converted to a signal linear to luminance with a secondary function. The secondary function described in the publication is represented by equation 1. The equation 1 is determined such that the XYZ three stimulus values of a nine-gradations color chip of an achromatic color are measured and the signal becomes linear to Y value as luminance.          (                                                                                                            R                    ′                                    =                                                            -                      1.773                                        +                                          0.1369                      ⁢                      R                                        +                                          0.0006568                      ⁢                                              R                        2                                                                                                                                                                                      G                    ′                                    =                                                            -                      0.1946                                        +                                          0.09309                      ⁢                      G                                        +                                          0.0008552                      ⁢                                              G                        2                                                                                                                                                                                      B                    ′                                    =                                                            -                      0.2366                                        +                                          0.07422                      ⁢                      B                                        +                                          0.001001                      ⁢                                              B                        2                                                                                                                                                      (            1            )                              
Subsequently, in procedure 3102, the XYZ three stimulus values are calculated from the luminance linear signal by a multiple regression matrix which is used up to at least a secondary term. Finally in procedure 3103, the spectral reflectance is estimated from the XYZ three stimulus values. The multiple regression matrix in the procedure 3102 needs to be predetermined. In order to determine the multiple regression matrix, skin is first photographed as a specific subject by the image input apparatus to obtain image data, and further skin color is measured with a colorimeter to obtain the XYZ three stimulus values. Subsequently, a matrix M for converting the image data to the XYZ three stimulus values is determined such that an error between the XYZ three stimulus values estimated by conversion and the XYZ three stimulus values measured by the calorimeter is minimized. To determine the estimating matrix in such a manner that the error between a predicted value and an actual measurement is minimized is referred to as multiple regression analysis and the estimating matrix determined in this manner is referred to as the multiple regression matrix. Assuming that a XYZ three stimulus values vector is T and image data vector is I, the multiple regression matrix is represented by equation 2. In the equation 2, RTI denotes a correlation matrix of T and I.M=RTIRII−1  (2)
Moreover, a dimension of the spectral reflectance in the procedure 3103 is as extremely large as 31 dimensions, and is difficult to estimate, even when a range of a visible light, for example, of 400 nm to 700 nm is sampled every 10 nm. Therefore, a method of performing a principal component analysis and representing a base having m-dimensions lower than 31-dimensions is used. Since a cumulative contribution ratio of a third principal component of the spectral reflectance of the skin as the subject is 99.5%, m=3 is sufficient, and a coefficient of the base can uniquely be obtained from the XYZ three stimulus values. In the aforementioned conventional characterization method, the subject is limited to the skin, and the matrix for estimating the XYZ three stimulus values from the image data is determined by the multiple regression analysis of the image data of the skin and the actually measured XYZ three stimulus values. Therefore, the XYZ three stimulus values of the skin can highly precisely be estimated in the matrix, but the XYZ three stimulus values of a subject other than the skin has an extremely large error.
Furthermore, in the color reproduction apparatus described in Japanese Patent Application Laid-Open No. 85952/1999, the matrix for obtaining the XYZ three stimulus values from the image data is derived as follows. First, the three stimulus values vector T and image data vector I can be represented by equation 3.                     (                                                            T                =                                                      E                    0                                    ⁢                  Xf                                                                                                        I                =                                                      E                    m                                    ⁢                  Sf                                                                                        (        3        )            
In the equation 3, E0 denotes a lighting matrix during observation, X denotes a matrix using a color matching function as a lateral vector, f denotes a spectral reflectance, Em denotes a lighting matrix during photographing, and S denotes a matrix using a spectral sensitivity of the image input apparatus as the lateral vector. When the equation 3 is assigned to the multiple regression matrix (equation 2), equation 4 is obtained. In the equation 4, Rff is a correlation matrix of the spectral reflectance of the subject. The correlation matrix of the spectral reflectance of the subject as a main constituting element of the input image is calculated beforehand, and assigned to Rff of equation 4, so that a matrix (equation 4 ) for estimating the XYZ three stimulus values from the image data can be obtained.M=E0XRffEmS(EmSRffEmS)−1  (4)
As described above, in the characterization method of the image input apparatus, the subject is limited, and the correlation matrix of the spectral reflectance of the limited subject is used to determine the matrix for estimating the XYZ three stimulus values from the image data. Therefore, when the XYZ three stimulus values of the image data of the subject other than the limited subject are estimated by the matrix, the error becomes extremely large.
Moreover, in the color simulation apparatus described in the Japanese Patent Application Laid-Open No. 233490/1997, lighting simulation is disclosed in which the image inputted from the image input apparatus is converted to a color under a desired light source, and then outputted onto a display. The procedure will be described with reference to FIG. 32. Similarly as the method described in the Japanese Patent Application Laid-Open No. 174631/1995, the principal component of the spectral reflectance is analyzed, and the reflectance is represented by the base having the m-dimensions lower than 31 dimensions. Subsequently, in procedure 3201, a base coefficient m-dimensional vector is estimated from the input image data by a neural network.
Next in procedure 3202, the spectral reflectance is calculated from the estimated m-dimensional vector. A desired light source vector is applied to the obtained spectral reflectance to obtain the XYZ three stimulus values, and a color property of a display is used to convert the values to a display drive signal. In the neural network, when the input data has a property similar to that of learning data, an appropriate spectral reflectance is estimated, but the error becomes extremely large with the input data which is not similar to the learning data. Therefore, this conventional example using the neural network can be said to be a method for enabling the estimate, only when the subject is limited. Any one of the aforementioned conventional methods comprises limiting the subject to one, determining the matrix or the neural network for estimating the spectral reflectance beforehand, and estimating the XYZ three stimulus values or the spectral reflectance of all pixels in the input image with one matrix or neural network.
However, the image to be actually photographed is rarely constituted of only the limited subject. For example, when the image of an upper part of a person's body is inputted, many of the pixels of the image are skin. Therefore, when the subject is limited to the skin, and the regression matrix for estimating the XYZ three stimulus values from the input image data is prepared beforehand, the XYZ three stimulus values of the skin can highly precisely be estimated by the regression matrix. However, when portions other than the skin, such as glasses, clothes, and hair are estimated by the regression matrix, the error disadvantageously becomes considerably large. To solve the problem, when the regression matrix is prepared from more subjects including the glasses and clothes without limiting the subject only to the skin, the precision of the estimated value of the subject other than the skin is raised as compared with use of the regression matrix prepared only for the skin. However, the precision is not very high. On the other hand, the estimate error of the skin which is essential becomes large as compared with the use of the aforementioned regression matrix. This is because the subjects different in statistical property such as the skin, eyeglasses, and clothes are included.
Moreover, when the subject is limited to the skin, the number of dimensions of the spectral reflectance can be lowered to three dimensions by the principal component analysis. However, when the spectral reflectance of more subjects is subjected to the principal component analysis without limiting the subject to the skin, the necessary dimension exceeds three dimensions. For example, in Journal of Optical Society America A, Vol. 3, No. 10, 1986, page 1673, “Evaluation of Linear Models of Surfaces Spectral Reflectance with Small Number of Parameters”, a fact that about six or eight dimensions at minimum are required for representing the spectral reflectance of an arbitrary subject is described. Therefore, in the image input apparatus whose number of bands is small in a range of 6 to 8, when the subject is arbitrary, the spectral reflectance cannot uniquely be calculated.
As described above, the method of calculating the XYZ three stimulus values or the spectral reflectance of the subject from the image data of the image input apparatus with the high precision is a problem which remains unsolved.